ON LOCAL COEFFICIENTS FOR NON-GENERIC REPRESENTATIONS OF SOME CLASSICAL GROUPS Solomon Friedberg and David Goldberg

This paper is concerned with representations of split orthogonal and quasi-split unitary groups over a nonarchimedean local field which are not generic, but which support a unique model of a different kind, the generalized Bessel model. The properties of the Bessel models under induction are studied, and an analogue of Rodier’s theorem concerning the induction of Whittaker models is proved for Bessel models which are minimal in a suitable sense. The holomorphicity in the induction parameter of the Bessel functional is established. Last, local coefficients are defined for each irreducible supercuspidal representation which carries a Bessel functional and also for a certain component of each representation parabolically induced from such a supercuspidal. Introduction. L-functions are a central object of study in representation theory and number theory. Over a global field, one has the Langlands Conjectures, which assert in particular the meromorphic continuation and functional equation of a class of Euler products. Over a local field one has additional conjectures due to Langlands, expressing the Plancherel measure arithmetically as the ratio of certain local L-functions and root numbers. In many cases these conjectures have been established by Shahidi [Shab,Shac, Shad], following a path laid out by Langlands [Lana]. The framework for Shahidi’s work is the study of Eisenstein series or their local analogues, induced representations. One knows the continuation of these Eisenstein series due to Langlands [Lanb]. Langlands also showed that the constant coefficients of the Eisenstein series may be expressed in terms of local intertwining operators which are almost everywhere quotients of certain L-functions. It remains to study these intertwining operators for the finite set of ‘bad’ places. If the inducing data is generic, that is, admits a Whittaker model, then Shahidi has succeeded in relating them to local L-functions. Thus the careful study of the Eisenstein series, both local and global, affords a proof of certain of the Langlands conjectures for these L-functions. The aim of this work is to suggest that the Langlands-Shahidi method may be extended beyond the generic spectrum by the use of other models. The Whittaker model is unique (an irreducible admissible representation admits at most one such model up to scalars). In this paper we study the properties of local representations of split orthogonal groups and quasi-split unitary groups which are not generic, but 1991 Mathematics Subject Classification. Primary 22E50, Secondary 22E35. Friedberg was partially supported by National Security Administration Grant MDA904-95-H1053. Goldberg was partially supported by National Science Foundation Fellowship DMS-9206246 and National Science Foundation Career Grant DMS-9501868. The research of both authors at MSRI was supported in part by National Science Foundation Grant DMS-9022140.